ChatGPT and other generative artificial intelligence tools are rising in popularity. If you have ever used these tools, you might have realised that you are revealing your thoughts (and possibly emotions) through your questions and interactions with the AI platforms. You can therefore imagine the huge amount of data these AI tools are gathering and the patterns that they are able to extract from the way we think. The impact of these business practices is crystal clear: a new AI economy is emerging through collecting, codifying, and monetising the patterns derived from our thoughts and feelings. Intrusions into our intimacy and cognition will be much greater than with existing social media and tech platforms.

The Transform Technology Summits start October 13th with Low-Code/No Code: Enabling Enterprise Agility. Open source technology has been a driving factor in many of the most innovative developments of the digital age, so it should come as no surprise that it has made its way into artificial intelligence as well. But with trust in AI's impact on the world still uncertain, the idea that open source tools, libraries, and communities are creating AI projects in the usual wild west fashion is creating yet more unease among some observers. Open source supporters, of course, reject these fears, arguing that there is just as little oversight into the corporate-dominated activities of closed platforms. In fact, open source can be more readily tracked and monitored because it is, well, open for all to see. And this leaves us with the same question that has bedeviled technology advances through the ages: Is it better to let these powerful tools grow and evolve as they will, or should we try to control them?

Apache Open Source Projects – Open source software has long been mistakenly considered inferior to proprietary software. But in the meantime many successful Apache Open Source projects could teach you better. They are often not only the big, whole solution, but can be used modularly for small problems and allow access to the know-how of many developers. Especially in the data science sector, many exciting projects based on the Python programming language have been established in recent years, which are built, maintained and continuously expanded by large, very active communities. In the meantime, these solutions have also been accepted in the business world.

In the past year or two, many companies have shared their data discovery platforms (the latest being Facebook's Nemo). Based on this list, we now know of more than 10 implementations. I haven't been paying much attention to these developments in data discovery and wanted to catch up. By the end of this, we'll learn about the key features that solve 80% of data discoverability problems. We'll also see how the platforms compare on these features, and take a closer look at open source solutions available.

Today's IoT use cases increasingly depend on performing analytics or updating machine learning algorithms in real time on huge amounts of device-generated data. If the data for patient monitoring, autonomous vehicles or predictive maintenance applications isn't ingested, processed and acted upon in real time, patients suffer, vehicles crash or systems fail. So how can businesses cost-effectively build a reliable platform for ingesting and responding to massive amounts of data at scale? Businesses can do so with a streaming platform and data storage system built on an open source software stack. Many of today's open source solutions have proven to be reliable across thousands of production deployments.

LinkedIn heavily relies on artificial intelligence to deliver content and create economic opportunities for its 575 million members. Following recent rapid advances of deep learning technologies, our AI engineers have started adopting deep neural networks in LinkedIn's relevance-driven products, including feeds and smart-replies. Many of these use cases are built on TensorFlow, a popular deep learning framework written by Google. In the beginning, our internal TensorFlow users ran the framework on small and unmanaged "bare metal" clusters. But we quickly realized the need to connect TensorFlow to the massive compute and storage power of our Hadoop-based big data platform.

Josh Poduska is a Senior Data Scientist in HPE's Big Data Software Group. Josh has 16 years of experience in the analytical sciences with an emphasis on machine learning and statistical applications. He spent the last five years focusing on advanced analytical solutions with MPP columnar databases. At HPE he is part of the Vertica team and uses Vertica and its machine learning library to help organizations solve their toughest data challenges. What are the main technical challenges you typically face in this era of Big Data Analytics?